Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Auto File Management Part 3: Install data-management tool as command line program #246

Open
wants to merge 37 commits into
base: main
Choose a base branch
from

Conversation

mabruzzo
Copy link
Collaborator

This PR is the 3rd in the sequence of PRs introducing automatic data-management. (Following #235 and #237). This is the last one in the sequence

To Recap:

  1. PR Auto File Management Part 1: Introducing a Datafile Management Tool #235 introduced all of the logic for the grdata cli tool. The tool was only invocable through pygrackle (via python -m pygrackle <args>…). The tool was only useful for testing pygrackle testing
  2. PR Auto File Management Part 2: Allow Grackle to search for automatically managed data files #237 introduced logic to the core grackle library to make it possible to access files that are fetched and managed by the tool.

This PR:

  • adds logic to install this grdata cli tool as a standalone command line script (in a bin directory next to the include and lib directories of a grackle install)
  • I converted one of the code-examples to make use of automatic datafile management. I'm tempted to convert all of the examples (so that the examples can all be invoked from arbitrary directories), but I wanted feedback on this point before doing that
  • documents how to use the grdata cli tool. I ended up updating a bunch of existing documentation to make things more clear (among other things, I moved the description of the code-examples & how to run them to a new, dedicated page)

I also added documentation and integrated the tool into the testing
framework.
We plan to eventually install the grdata tool as a standalone command
line program. Essentially the build-system will perform some
substitutions (the CMake build system uses CMake's built-in
``configure_file`` command while the classic build system uses the
analogous ``configure_file.py`` script)

This commit introduces a few minor tweaks to grdata.py so that it can
more easily be consumed by the ``configure_file.py`` script.
- The ``configure_file.py`` script, itself, will ultimately require a
  few more tweaks so that it doesn't report occurences of python's
  decorator-syntax as errors
- However, this commit minimizes the number of required changes
Among other things, we started using picohash and using the functions in
os_utils.ch
The file registry is encoded in the autogenerated file_registry.h file
that is produced from file_registry.h.in.

To get this to work properly for the Makefile build-system, I needed to
add a new feature to ``configure_file.py``. In detail:

* ``configure_file.py`` already provided the option to replace a
  variable in a template file with multiple lines of content read from
  an external file. We assumed that this option would only be used for
  formatting multiline strings in printf statements. Consequently, the
  machinery would replace any new-line characters encountered in the
  external file with the "\n" escape-sequence used in C strings to
  represent a new-line.

* I added simply added the option to ``configure_file.py`` to do the
  same thing WITHOUT escaping new-line characters.
This commit makes the following changes to the CMake build system:
- it now creates the grdata executable and installs it
- it exports the program's location as a variable in grackle.pc
- it exposes the program's location through a custom CMake command
  (that custom CMake command will be revisited in future commits)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant